\chapter{Evaluation}
\label{chap:eval}
\section{Case Study - Delta Calculator}
\label{sec:evaldel}
The first case study was on the peer review team project contribution calculator, also known as the delta calculator. It takes in .csv files from students working together in a team, and based upon the level that each student has been rated by the fellow team-mates, it returns a number. Negative numbers imply they did less work than the average person, and positive numbers imply the opposite. These numbers are used to calculate the grade of the student relative to their team's overall grade.

A python script was used in order to generate .csv files, containing randomized scores, to run on this application. See Appendix~\ref{app:gen} - \nameref{app:gen} for the script used to generate them. It takes a number as input, and generates that many .csv files for one team containing five people.

The library was added to the application, and was run. The log4j.properties file was modified to include the aspects, as shown in the example file in Appendix~\ref{app:aop} - \nameref{app:aop}. The aop.xml file was placed into the META-INF folder, within the source. It was configured to be \lstinline!within(uk.ac.gla.socs.peerreview..*)! with \lstinline!include within="uk.ac.gla.socs.peerreview.*"!. The application's build file was modified to run the library's build file and use the aspectjweaver as its agent.

The TimerAspect was run against a set of five generated .csv files. This is within reasonable usage of the system, so the system's output should be predictable. The first 100 lines of the output from this can be found in Appendix~\ref{app:atd} - \nameref{app:atd}. The total runtime of the application, with the library, is around a seventh of a second (not including the time in which the library's code was running). This almost identical to the amount of time found when not using the library. I ran this test several times, with and without the library, and got similar results. This confirms that the first non-functional requirement had been fulfilled. This also proved the working order of functional requirements 1, 2, 4, and 8.

The application was then run with the MethodTouchesAspect. The output of this can be found in Appendix~\ref{app:amd} - \nameref{app:amd}. This data was used to draw a graph, which can be seen in Figure~\ref{fig:exg}, in the previous chapter. This proves the working order of functional requirements 3, 6, 7 (to a degree), and 9.

\begin{figure}[H]
	\caption{A graph showing the number of uses of each method from the Delta Calculator Case Study}
	\label{fig:mtd500}
	\centering
		\includegraphics[width=\textwidth]{images/MethodTouchesDelta500.png}
\end{figure}

The system was then tested with 500 generated .csv files. This is outside the expected input for the system, and more likely to show if there are any issues with the scalability of the system. The resulting graph from this can be seen in Figure~\ref{fig:mtd500}. Upon comparison of the two graphs, we can see the the number of methods calls for each method is proportional the the amount of data inputted, with the exception of \lstinline!Matrix.getRow(..)!, which is called five times (the number of students in the group), \lstinline!Coordinate.equals(..)! , which is called twice in the second case, and does not seem to appear in the first case, and \lstinline!Matrix.getColumnLabels!, \lstinline!Matrix.getRowLabels()!, \lstinline!Main.main(..)! and \lstinline!Matrix.getColumnLabels!, which are all only called once.

Non-functional requirements 3 and 4 (and arguable 2) are subjective, and hence much more difficult to evaluate. However, I would argue that the output from the system is understandable in the case of the graph and the log file.
\newpage
\section{Case Study - Password Cracker By Profile}
The second case study was on the Password Cracking By Profile (pcbp) application. It attempts to break a password using a particular character set, given a sequence string. This evaluation is different from the delta calculator case study as the system will be tested against the JUnit tests of the application, rather than the main function. This required similar modifications of the build.xml file as the previous case study did: setting the agent, building the aspectjlib, etc. The system was run the system on its CharSetTest tests using the TimerAspect. This resulted in a large aspect.csv file (over 11 MB). The first 100 lines of it can be viewed in Appendix~\ref{app:pcbpt} - \nameref{app:pcbpt}.

\begin{figure}[H]
	\caption{A graph showing the number of uses of each method from the Delta Calculator Case Study}
	\label{fig:mtpcbp}
	\centering
		\includegraphics[width=\textwidth]{images/MethodTouchesPCBP.png}
\end{figure}

The system was then run under the same conditions using the MethodTouchesAspect. The output file can be viewed in Appendix~\ref{app:pcbpm} - \nameref{app:pcbpm}, and the output image is shown in Figure~\ref{fig:mtpcbp}. This graph, as well as the output file, shows an interesting characteristic of the library. Normally the method touches are ordered, in descending order. This is technically still true for this, however they are ordered on a ``per test basis''. This means that each test is ordered internally. Because of this, the graph is rather unreadable. The logs are much more useful for this.

\section{Evaluation Limitations}
\label{sec:evallim}
Unfortunately, this library was unable to be tested against a multi-threaded system. Because of this, there is no way to be sure as to whether function requirement 10 has been met, or not. The functionality certainly exists for the library to work for it, but it has not been tested or evaluated.

Another issue encountered was the inability to test the system against a system that used Spring, or an aspect-oriented application. It is uncertain how the system would react to these applications, especially if they were not also configured for load-time weaving.

\chapter{Conclusion}
\label{chap:conc}
\section{Reflection on Design}
The overall architecture of the library seems to have been a success, however it could be improved by higher levels of abstraction. Ideally, the MasterTracker should only be called by a single aspect, and the enum passed to the MasterTracker could be defined within the configuration file, meaning that the user would not need to comment out the concrete aspects, and could instead change the aspect simply by changing a variable name.

The design of the logging mechanism has the limitation of requiring a path to the log4j.properties file. If there were some way to specify the location of the log4j.properties file within the configuration, that would be an ideal change to the system.

As for the Logging Package, overall the interactions between the MasterTracker and the ThreadTrackers have been good, however due to the \nameref{sec:evallim} involving the lack of multi-threaded applications to test the system against, it is uncertain as to whether this is optimal.

The Visual package was not fully implemented, but it was not a priority to implement. The framework, the aspects, and the logging were the priorities of the library, so the decision to focus on that seems to have been the right one.

\section{Extensibility}
The first, and most obvious, extension to the system is the inclusion of more aspects. Aspects that measure the size of input and output to methods would be relatively simple to implement, as would an aspect that outputs the stack depth at various points throughout the application's runtime.

Another potential extension to the library could be the full implementation the Visual package. The open-source library used by the Grapher, JFreeChart, supports the real-time updating of a .csv file, so it could be used to track the method calls of an application on a thread by thread basis. This would be especially useful for web applications.

\section{Final Thoughts}
Despite the limitations of the evaluation, and the potential improvements to the design, the overall system is a fully functioning library. It can be used by real users, be they independent developers or employees of a large corporation. Given that providing a system to solve the problem of the real-time monitoring of non-functional behaviour was the initial goal of this project, the aspectjlib package was a success.
